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Abstract 

Background: Meta-analyses are particularly vulnerable to the effects of publication bias. Despite methodologists' 
best efforts to locate all evidence for a given topic the most comprehensive searches are likely to miss unpublished 
studies and studies that are published in the gray literature only. If the results of the missing studies differ 
systematically from the published ones, a meta-analysis will be biased with an inaccurate assessment of the 
intervention's effects. 

As part of the OPEN project (www.open-project.eu) we will conduct a systematic review with the following 
objectives: 

■ To assess the impact of studies that are not published or published in the gray literature on pooled effect 
estimates in meta-analyses (quantitative measure). 

■ To assess whether the inclusion of unpublished studies or studies published in the gray literature leads to 
different conclusions in meta-analyses (qualitative measure). 

Methods/Design: Inclusion criteria: Methodological research projects of a cohort of meta-analyses which compare 
the effect of the inclusion or exclusion of unpublished studies or studies published in the gray literature. 
Literature search: To identify relevant research projects we will conduct electronic searches in Medline, Embase and 
The Cochrane Library; check reference lists; and contact experts. 

Outcomes: 1) The extent to which the effect estimate in a meta-analyses changes with the inclusion or exclusion of 
studies that were not published or published in the gray literature; and 2) the extent to which the inclusion of 
unpublished studies impacts the meta-analyses' conclusions. 

Data collection: Information will be collected on the area of health care; the number of meta-analyses included in 
the methodological research project; the number of studies included in the meta-analyses; the number of study 
participants; the number and type of unpublished studies; studies published in the gray literature and published 
studies; the sources used to retrieve studies that are unpublished, published in the gray literature, or commercially 
published; and the validity of the methodological research project. 

Data synthesis: Data synthesis will involve descriptive and statistical summaries of the findings of the included 
methodological research projects. 

Discussion: Results are expected to be publicly available in the middle of 2013. 
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Background 

A meta-analyses as part of a systematic review aims to 
provide a thorough, comprehensive and unbiased ac- 
count of the literature [1,2]. However, potentially im- 
portant studies could be missing from a meta-analysis 
because of selective publication and inadequate dissem- 
ination of results. Despite methodologists' best efforts 
to locate all eligible evidence for a given topic the 
most comprehensive searches are likely to miss un- 
published studies and studies that are not commer- 
cially published and, therefore, are not indexed in 
respective databases (so called gray literature, such as 
conference abstract, dissertations, policy documents, 
book chapters). If the results from missing studies 
differ systematically from the published data, a meta- 
analysis may become biased with an inaccurate as- 
sessment of the interventions effects. For instance, 
positive, significant findings are more likely to be 
published than non-significant findings, and a meta- 
analysis which is based mainly on published literature 
may end up overestimating the efficacy of the inter- 
vention [3-5]. 

However, the impact of gray literature and unpub- 
lished studies on the conclusions of meta-analyses has 
not been comprehensively clarified. For example, there 
is some evidence that suggests that published random- 
ized controlled trials (RCTs) tend to be larger and show 
an overall greater treatment effect than gray trials [6]. 
But the identification of relevant unpublished studies or 
studies published in the gray literature and their 
inclusion in meta-analyses can be particularly time- 
consuming and challenging. There is also some contro- 
versy as to whether unpublished studies and studies 
published in the gray literature should be included in 
meta-analyses because they might be incomplete and 
their methodological quality (validity) can be difficult to 
assess. A publication by Cook and colleagues in 1993 
showed that 78% of authors of meta-analyses felt that 
unpublished studies should be included in meta-analyses 
compared to only 47% of journal editors [7]. Therefore, 
research is needed to help assess the potential implica- 
tions for reviewers of not including gray literature and 
unpublished studies in meta-analyses of health care 
interventions. 

Objectives 

In terms of the above mentioned controversies regarding 
the inclusion of unpublished studies and studies pub- 
lished in the gray literature on the results of meta- 
analyses, we will conduct a systematic review with the 
following objectives: 

■ To assess the impact of studies that were not 
published or only published in the gray literature on 



pooled effect estimates in meta-analyses (quantitative 
measure) 

■ To assess whether the inclusion of unpublished 
studies or studies published in the gray literature 
impacts the conclusions of meta-analyses (qualitative 
measure) 

This systematic review will be part of the OPEN Pro- 
ject (To Overcome failure to Publish nEgative fiNdings) 
which was developed with the goal of elucidating the 
scope of non-publication of studies through a series of 
systematic reviews. In an earlier issue of this journal 
('Systematic Reviews'), our group has already published a 
protocol for a systematic review which evaluates the ex- 
tent of non-publication of research studies, which were 
approved by ethics committees, registered in trial regis- 
tries or presented as conference abstracts [8]. 

Methods/Design 

Search methods for identification of methodological 
research projects 

To identify the relevant research evidence we will con- 
duct electronic literature searches in the following data- 
bases: Ovid Medline (1946 to present), Ovid Medline 
Daily Update, Ovid Medline in process & other non- 
indexed citations, Ovid Embase (1980 to present), The 
Cochrane Library (most current issue) and Web of 
Science, No language restrictions will be applied. 

In addition, the bibliographies of any eligible articles 
identified will be checked for additional references and 
citation searches will be carried out for all included ref- 
erences using ISI Web of Knowledge. 

A search strategy for the electronic literature search in 
Ovid Medline has already been designed with the sup- 
port of a librarian/information specialist. This strategy 
was translated as appropriate for the other databases 
(for the full search strategies see Appendix A). In 
addition, we will contact various experts in the field for 
further eligible studies. 

Data collection and analyses 

Selection of methodological research projects 

A methodological research project will be considered eli- 
gible for inclusion in this systematic review if it reviews 
a cohort of meta-analyses (that is, more than one meta- 
analyses) that: 

■ compare pooled effect estimates of meta-analyses of 
health care interventions according to publication 
status (that is, published versus unpublished studies 
or gray literature) or 

■ examine whether the inclusion of unpublished studies 
or gray literature impacts the overall findings or 
conclusions of a meta-analyses 
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We will consider published' articles to be manuscripts 
that appeared in peer-reviewed journals. Our working 
definition of gray literature will correspond to the 
definitions used by the authors of eligible methodo- 
logical research projects and which also conforms to the 
definition of 'gray literature' described earlier in this 
protocol (see Background). A meta-analysis is defined as 
the calculation of a summary estimate of treatment 
effect by pooling the results of two or more studies. 

Data extraction 

A specifically designed data extraction form will be 
developed and two reviewers will independently extract 
all relevant data from eligible methodological research 
projects. The following information will be collected: 

1. Characteristics of the methodological research 
project 

a. Baseline data (for example, author names, 
affiliation, language and year of publication, 
funding, type of report (for example, full 
publication, abstract)) 

b. Area of health care/medical specialty 

c. Number of meta-analyses included 

2. Characteristics of the meta-analyses included in the 
methodological research project 

a. Type of meta-analyses (for example, individual 
patient data meta-analyses) 

b. Number of studies included in meta-analyses 
(overall, median, range) 

c. Number of participants included in meta-analyses 
(overall, median, range) 

d. Main purpose of meta-analyses (efficacy versus 
safety) 

e. Source used to retrieve unpublished studies, 
studies published in the gray literature and 
published studies 



studies 

d. Type of unpublished studies (for example, 
RCTs, observational studies), studies 
published in the gray literature (for example, 
abstracts, dissertation, letter, book chapters) 
and published studies (for example, RCTs, 
observational studies) 



e. Year of publication of unpublished studies, 
studies published in the gray literature and 
published studies 

f. Language and country of unpublished studies, 
studies published in the gray literature and 
published studies 

g. Funding source of unpublished studies, studies 
published in the gray literature and published 
studies 

h. Type of data source in which gray, unpublished 
and published studies were identified 

i. Methodological quality (for example, blinding, 
follow-up time, sample size calculation) of 
unpublished studies, studies published in 

the gray literature and published studies 
(this aspect can only be evaluated if the 
methodological research project provides 
enough information) 

Assessment of validity 

We will systematically consider the validity and genera- 
lizability of the identified evidence provided by each of 
the methodological research projects by evaluating the 
following aspects: 

1. Internal validity: 

a. Role of confounding factors: The results of 
published studies may differ from those of 
unpublished studies because of factors other than 
publication status, such as study design, type of 
participants, characteristics of the intervention, 
and methodological quality; in this context, did 
the researcher of the meta-analyses select 
comparison groups that were matched (for 
example, did the unpublished studies or studies 
published in the gray literature share similar 
aims, designs, and sample sizes as the published 
ones)?; if not, were suitable adjustments for 
potentially confounding factors made? 

b. Definition of publication status: Are explicit 
criteria given to categorize or define unpublished 
studies, studies published in the gray literature 
and published studies? 

c. Selection process: Are search criteria given to 
identify unpublished studies, studies published in 
the gray literature and published studies? 

External validity (generalizability): 

a. Did the researcher of the methodological research 
project select a broad-ranging sample of 
meta-analyses that reflect the current 
literature in the field of interest (for example, 
in terms of size, diversity of topic)? 

b. How was the sample determined (for example, 
random sample)? 



3. Characteristics of the studies included in the meta- 
analyses 

a. Number of unpublished studies, studies published 
in the gray literature and published studies 

b. Number of participant in unpublished studies, in 
studies published in the gray literature and in 
published studies 

c. Number of statistically significant positive or 
negative unpublished studies, studies 
published in the gray literature and published 2. 
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c. Did two researchers carry out data extraction 
independently? 

d. Did the researchers provide a complete dataset 
(regarding the characteristics of the 
methodological research project and included 
meta-analyses)? 

Outcome measures 

1. The extent to which the effect estimate in a 
meta-analyses changes with the inclusion or 
exclusion of unpublished studies and gray 
literature (quantitative measurement,). If possible, 
we will calculate a ratio of risks or odds ratios 
between the results of unpublished studies and 
studies published in the gray literature and the 
results of published studies and estimate the 
percentage change (pooled risk ratio from 
unpublished studies and gray literature divided by 
pooled risk ratio from published studies). A 
weighted pooled overall estimate will be 
calculated taking into account number of studies, 
participants and events. 

2. The impact of the inclusion of unpublished studies 
or studies published in the gray literature on 
conclusions of meta-analyses (qualitative 
measurement). The impact will be estimated by 
calculating the proportion of meta-analyses which 
show a change in their conclusions according to 
publication status of the included studies; 
categorization will be as follows: 

a. Change from negatively significant to positively 
significant 

b. Change from inconclusive to positively significant 

c. Change from positively significant to inconclusive 

d. Change from negatively significant to 
inconclusive 

e. Change from inconclusive to negatively 
significant 

f. Change from positively significant to negatively 
significant 

g. Change from not clinical relevant to clinical 
relevant 

h. Change from not clinical relevant to inconclusive 

i. Change from clinical relevant to inconclusive 
j. Change from clinical relevant to not clinical 

relevant 

k. Change from inconclusive to clinical relevant 
1. Change from inconclusive to not clinical relevant 

Significance and clinical relevance will be defined 
according to the definitions provided in the methodo- 
logical research project. 



Unit of analyses issues 

The anticipated unit of analyses is the meta-analyses in- 
cluded in the methodological research project. 

Assessment of heterogeneity 

Heterogeneity for pooled outcome measures will be 
assessed by standard methods including Chi 2 -test and 
calculation of the I 2 value [9]. 

Assessment of reporting biases 

Funnel plots will be used to assess the association be- 
tween point estimates of log odds ratio (a measure of 
extent of association between meta-analyses' characteris- 
tics and change in summary estimates) and a measure of 
precision if more than ten methodological research pro- 
jects provide necessary information. Funnel plots will be 
visually assessed and appropriate formal statistical tests 
following recommendations formulated by Sterne et al. 
will be used to test for asymmetry [5]. In the instance of 
suspected reporting bias authors will be contacted. 

Data synthesis 

Data synthesis will involve a combination of descriptive 
and statistical summaries of the impact of the inclusion 
or exclusion of unpublished studies and gray literature 
on the results of meta-analyses (identified by methodo- 
logical research projects). 

The decision on whether or not to combine the results 
of the included methodological research projects will 
depend on the assessment of heterogeneity. Where 
methodological research projects will be judged to be 
sufficiently homogenous in their design a meta-analyses 
of these research projects will be carried out. The esti- 
mated ratios of unpublished and published in the gray 
literature only versus published treatment effects gener- 
ated from each methodological evaluation will then be 
used to summarize the overall difference in risk ratios 
between unpublished and published in the gray literature 
only and published studies. The 95% confidence interval 
for the combined effect will be estimated using a ran- 
dom effects model. 

Subgroup analyses and investigation of heterogeneity 

The following subgroup analyses are planned: 

1. On the level of the methodological research project 

a. Number of meta-analyses included in the 
methodological research project 

b. Number of participants included in the 
methodological research project 

2. On the level of the meta-analyses 

a. Number of studies (unpublished studies versus 
studies published in the gray literature versus 
both) 
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b. Number of participants included in studies 
(unpublished studies versus studies published in 
the gray literature versus both) 

c. Design of studies (unpublished studies versus 
studies published in the gray literature versus 
both) 

d. Source of database: gray literature (for example, 
conference abstracts or research letters) 
published in an easily accessible database versus 
unpublished studies for which immense efforts 
are required to be identified (for example, contact 
with pharmaceutical industry) 

e. Type of research work (drug versus non-drug 
studies, clinical research versus basic research) 

f. Area of health care 

Sensitivity analyses 

No sensitivity analyses are planned. However, should the 
instance arise where a methodological research project 
with doubtful eligibility is identified (as determined by 
individual review and analyses of the validity of the 
methodological research project), sensitivity analyses 
may be undertaken. Possible sensitivity analyses may be 
based on the following: 

■ Methodological quality/validity of the methodological 
research project; only methodological research projects 
with low risk of bias will be considered. A methodological 
research project will be of high risk of bias when the 
external validity and/or the validity of the included meta- 
analyses are doubtful. 

Discussion 

This systematic review seeks to comprehensively synthe- 
size the growing body of research that is related to the 
impact of including unpublished studies and studies 
published in the gray literature in meta-analyses. By 
considering multiple characteristics and potential con- 
founders related to unpublished studies and studies 
published in the gray literature, we hope to identify suffi- 
cient evidence to conclude whether (or to what extent) 
inclusion of unpublished studies and studies published 
in the gray literature has an impact on the pooled effect 
estimates and the conclusions from a meta-analyses. The 
findings, including risk factors for unpublished studies 
and studies published in the gray literature, will have 
important implications for researchers conducting meta- 
analyses since they need to be informed about the im- 
pact and extent of (not) including unpublished and gray 
studies in meta-analyses. In addition, this systematic re- 
view in combination with the results of other systematic 
reviews that are part of the OPEN Project will serve to 
raise awareness about the impact of publication bias and 
the complexity of this issue. These reviews will also 



serve as a foundation for a recommendations workshop 
which will enable key members of the biomedical re- 
search community (for example, funders, research ethics 
committees, and journal editors) to develop future pol- 
icies and guidelines to lessen the frequency of non- 
publication and related biases. 

We acknowledge that more than half of all systematic re- 
views do not involve meta-analysis in their analyses. Des- 
pite the fact that our main outcomes focus on the impact 
of unpublished and gray studies on pooled effect estimates 
in meta-analyses, our findings will also be valuable for sys- 
tematic reviews. It is obvious, if we find a statistical differ- 
ence in the pooled effect estimates in meta-analyses, it is 
also likely that gray and unpublished literature impacts 
descriptive results of systematic reviews. Beside effect esti- 
mates, we will also evaluate differences in the methodo- 
logical quality and study characteristics (such as number of 
participants, language or methodological quality) between 
unpublished, gray and published studies. These results will 
also be valuable for systematic reviews to appraise the 
potential impact of publication bias. 

Appendix A 

Search Strategy for OvidSP MEDLINE (search strategy will 
be adapted for other databases) 

Line 10 to 14 and 16 to 35 are not shown because they are 
part of the search strategy for our 1 st systematic review [8] 
and, therefore, have no consequence for this search. 

1. exp Publishing/sn 

2. ''publishing/ 

3. publication bias/ 

4. selection bias/ 

5. exp manuscripts as topic/ 

6. ((data or finding? or information or evidence or study 
or studies or trial? or paper? or article? or report* or 
literature or work or manuscript? or abstract* or 
result?) adj6 (unpublish* or un-publish* or unreport* 
or un-report* or nonpublish* or non-publish* or 
nonpublicat* or non-publicat* or (publication? adj3 
rate?) or "not publish*") ).ti,ab. 

7. (underreport* or under-report* or selective report* or 
selective publish* or selective publicat* or (final* adj2 
(report* or publish* or publicat* or manuscript? or 
paper? or article?)) or (full? adj2 (report* or publish* 
or publicat* or manuscript? or paper? or article?)) or 
(subsequent* adj2 (report* or article? or paper? or 
publi* or manuscript?)) or (sub-sequent* adj2 (report? 
or article? or paper? or publi* or manuscript?)) or 
(complete* adj2 (report* or article? or paper? or 
publish* or publicat* or manuscript?))).ti,ab. 

8. (bias* adj3 (publish* or publicat*)).ti,ab. 

9. or/1-8 

15. exp animals/ not humans/ 
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36. 
37. 

38. 
39 
40. 
41. 



42. 
43. 
44. 
45. 
46. 
47. 
48. 



49. 



50. 



51. 



52. 



53. 
54. 
55. 
56. 



meta-analysis as topic/ 

Guidelines as Topic/ or Practice Guidelines as 
Topic/ 

exp Clinical Trials as Topic/ 
meta-analysis.pt. 

(guideline or practice guideline). pt. 
(guideline? or metaanaly* or meta-analy* or 
metanaly* or meta-synthe* or metasynthe* or 
meta-regressi* or metaregressi*).ti,ab. 
(systematic* adj3 (review* or overview*)). ti,ab. 
exp Technology Assessment, Biomedical/ 
(health technology assessment? or HTA).ti,ab. 
or/36-44 
9 and 45 

46 not 15 

((implication? or impact? or influenc* or effect? or 
differen*) adj6 (publication bias* or unpublish* or 
un-publish* or unreport* or un-report* or 
nonpublish* or non-publish* or nonpublicat* or 
non-publicat* or "not publish*")).ti,ab. 
((implication? or impact? or influenc* or effect? or 
differen*) adj6 (selective report* or selective publish* 
or selective publicat* or (final* adj2 (report* or 
publish* or publicat* or manuscript? or paper? 
or article?)) or (full? adj2 (report* or publish* or 
publicat* or manuscript? or paper? or article?)) 
or (subsequent* adj2 (report* or article? or paper? or 
publi* or manuscript?)) or (sub-sequent* adj2 
(report? or article? or paper? or publi* or 
manuscript?)) or (complete* adj2 (report* or article? 
or paper? or publish* or publicat* or manuscript?)))). 
ti,ab. 

((unpublish* or un-publish* or unreport* or un- 
report* or nonpublish* or non-publish* or 
nonpublicat* or non-publicat* or "not publish*") 
adj6 publish*).ti,ab. 

(underreport* or under-report* or selective report* or 
selective publish* or selective publicat* or (final* and 
(report* or publish* or publicat* or manuscript? or 
paper? or article?)) or (full? and (report* or publish* 
or publicat* or manuscript? or paper? or article?)) or 
(subsequent* and (report* or article? or paper? or 
publi* or manuscript?)) or (sub-sequent* and (report? 
or article? or paper? or publi* or manuscript?)) or 
(complete* and (report* or article? or paper? or 
publish* or publicat* or manuscript?))). ti. 
(unpublish* or un-publish* or unreport* or un- 
report* or nonpublish* or non-publish* or 
nonpublicat* or non-publicat* or "not publish*" or 
bias*).ti. 
or/48-52 

47 and 53 

47 and (3 or 4) 
47 and (36 or 37) 



57. 
58. 
59. 

60. 
61. 
62. 
63. 



64. 
65. 
66. 
67. 
68. 
69. 
70. 



56 and (6 or 7) 
"meta-analysis as topic/ 

"Guidelines as Topic/ or *Practice Guidelines as 
Topic/ 

47 and (58 or 59) 
(6 or 7) and (3 or 8) 
54 or 55 or 57 or 60 or 61 
(unpublish* or un-publish* or unreport* or un- 
report* or nonpublish* or non-publish* or non- 
publicat* or non-publicat* or "not publish*") .ti,ab. 
7 or 63 
62 and 64 
36 or 37 
3 and 66 
65 or 67 
68 not 15 

remove duplicates from 69 



Appendix B 

OPEN Consortium 

Table 1 shows OPEN Consortium. 



Table 1 OPEN Consortium 



Contributor Participating Institution 



Antes, Gerd German Cochrane Center, Institute of Medical Biometry 
and Medical Informatics University Medical Center, 
Freiburg, Germany 

Bassler, Dirk Center for Pediatric Clinical Studies, University Medical 
Center Tuebingen, Germany 

Bertele, Vittorio Department of Epidemiology, Mario Negri Institute for 
Pharmacological Research, Italy 

Bonfill, Xavier The Clinical Epidemiology & Public Health Department 
at the Hospital de la Santa Creu i Sant Pau, Spain 

Bouesseau, World Health Organization, Geneva, Switzerland 
Marie-Charlotte 

Boutron, INSERM U738 research unit, Paris Descartes University, 

Isabelle Paris, France 

Gallus, Silvano Department of Epidemiology, Mario Negri Institute for 
Pharmacological Research, Italy 

Garattini, Silvio Department of Epidemiology, Mario Negri Institute for 
Pharmacological Research, Italy 

Ghassan, Karam World Health Organization, Geneva, Switzerland 

La Vecchia, Department of Epidemiology, Mario Negri Institute for 
Carlo Pharmacological Research, Italy 

Lang, Britta German Cochrane Center, Institute of Medical Biometry 
and Medical Informatics University Medical Center, 
Freiburg, Germany 

Littmann, CELLS (Centre for Ethics and Law in Life Sciences), 

Jasper Hannover Medical Scholl, Hannover, Germany 

Kleijnen, Jos Kleijnen Systematic Reviews Ltd., York, United Kingdom 

Kulig, Michael Federal Joint Committee, Berlin, Germany 

Malicki, Mario University of Split School of Medicine, Split, Croatia 

Marusic, Ana University of Split School of Medicine, Split, Croatia 
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Table 1 OPEN Consortium (Continued) 



Meerpohl, German Cochrane Center, Institute of Medical Biometry 
Joerg and Medical Informatics University Medical Center, 

Freiburg, Germany 



Mueller, 

Katharina 

Felicitas 



Center for Pediatric Clinical Studies, University 
Medical Center Tuebingen, Germany 



Pardo, Hector The Clinical Epidemiology & Public Health Department 
at the Hospital de la Santa Creu i Sant Pau, Spain 



Perleth, 
Matthias 

Ravaud, 
Philippe 



Federal Joint Committee, Berlin, Germany 



INSERM U738 research unit, Paris Descartes University, 
Paris, France 



Reis, Andreas World Health Organization, Geneva, Switzerland 

Schmucker, German Cochrane Center, Institute of Medical Biometry 
Christine and Medical Informatics University Medical Center, 

Freiburg, Germany 

Schwarzer, German Cochrane Center, Institute of Medical Biometry 
Guido and Medical Informatics University Medical Center, 

Freiburg, Germany 

Strech, Daniel CELLS (Centre for Ethics and Law in Life Sciences), 
Hannover Medical Scholl, Hannover, Germany 

Trinquart, INSERM U738 research unit, Paris Descartes University, 

Ludovic Paris, France 

Urrutia, Gerard The Clinical Epidemiology & Public Health Department 
at the Hospital de la Santa Creu i Sant Pau, Spain 

Von Elm, Erik German Cochrane Center, Institute of Medical Biometry 
and Medical Informatics University Medical Center, 
Freiburg, Germany 



Wager, 
Elizabeth 



Sideview, Princes Risborough, United Kingdom 
Federal Joint Committee, Berlin, Germany 
Wolff, Robert Kleijnen Systematic Reviews Ltd., York, United Kingdom 



Wieland, 
Alexandra 



Center Tuebingen, 72070 Tuebingen, Germany. Cochrane Switzerland, 
IUMSP, University Hospital Lausanne, 1005 Lausanne, Switzerland. 
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